Add publication monitoring and revise the frontier panel#75
Conversation
|
Warning Review limit reached
Next review available in: 52 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe PR replaces the provisional 1,024-token reasoning-disabled lane with a frozen 4,096-token native-reasoning panel, adds exact route and reasoning validation, introduces publication-run status and smoke-artifact reuse, and updates evidence, documentation, tests, and leaderboard metadata. ChangesSOTA v2 publication lane
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related issues
Possibly related PRs
Sequence Diagram(s)sequenceDiagram
participant Registry
participant PublicationRunner
participant SmokeManifest
participant Leaderboard
Registry->>PublicationRunner: supplies frozen routes and reasoning policies
PublicationRunner->>SmokeManifest: records accepted route and telemetry metadata
SmokeManifest->>Leaderboard: validates publication identity and smoke state
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 10
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@config/sota_v2_models.json`:
- Around line 161-178: Remove the expiring route entry identified by id
openrouter-tencent-hy3-free-novita, or replace it with a stable, smoke-tested
route that remains valid beyond the freeze. Do not retain this route unless the
publication process is explicitly bounded before its catalog_expiration_date.
- Around line 14-15: The frozen model panel in the selection_policy and models
registry must contain twelve models: five proprietary and seven open-weight.
Update the model entries and associated required_smokes coverage so the declared
cohort, smoke completeness, cost planning, and leaderboard eligibility
consistently use the required five-plus-seven composition.
In `@docs/PUBLISH_READINESS.md`:
- Around line 13-17: Synchronize the frozen ten-model phase-one state across
docs/PUBLISH_READINESS.md lines 13-17, 30-32, 136-165, and 410, marking fresh
4,096-token smokes, verification, cost-plan, and lane-freeze work as complete
and removing provisional or retired-plan wording. Update
docs/output_budget_sweep.md lines 3-7 to call the current policy frozen, and
lines 62-65 to remove excluded Kimi K3 from the frozen lane’s reasoning
configuration.
In `@results/analysis/output-budget-cost-estimate.json`:
- Around line 127-136: Regenerate the runtime evidence in the output-budget cost
estimate from the accepted 4,096-token smoke observations, replacing the legacy
1,024-token source and updating the observation timestamp. Populate
observed_api_seconds_per_decision_by_model for all ten models in the accepted
ledger, rerun the estimator, and update the runtime status from
pending-smoke-telemetry to the resulting final evidence status.
In `@scripts/run_publication_matrix.py`:
- Around line 535-540: Keep the smoke-row state logic in
scripts/run_publication_matrix.py lines 535-540 based on current-run,
current-cap raw/checkpoint artifacts rather than manifest acceptance; expose
manifest acceptance separately or validate its artifact path/hash and cap before
using it. In tests/test_publication_runner.py lines 642-646, key the reservation
with cell.cap_label and assert the row-level reservation. In
tests/test_publication_runner.py lines 673-676, write raw artifacts under the
current 4,096-token stem so the monitor reads them.
- Around line 787-790: Update the reasoning-token validation around the
reasoning_policy checks to require reasoning_tokens to be a non-negative
integer, explicitly excluding booleans. Apply this validation to the
mandatory-minimum telemetry check and ensure negative values are rejected
wherever reasoning telemetry is validated, while preserving the existing
disabled-policy handling.
- Around line 995-999: Update the smoke reuse branch in the publication matrix
flow to skip reuse when args.preflight_only is enabled, ensuring that
_validate_openrouter_endpoint still runs for explicit live endpoint preflight
while preserving normal reuse behavior otherwise.
- Around line 87-100: Update the reasoning-policy validation around policy and
effort checks: require mandatory-minimum models to provide a non-null
reasoning_effort and matching OPENROUTER_REASONING_EFFORT, and reject disabled
models when fixed_options contains any reasoning-effort setting in addition to
enforcing disabled reasoning. Preserve the existing unknown-policy validation.
In `@web/scripts/build_leaderboard.py`:
- Around line 116-125: Update the upstream provider handling in the
expected_options construction to access registered["upstream_provider_slug"]
safely via the existing optional-value pattern, while preserving the current
OPENROUTER_PROVIDER_ONLY validation behavior when the slug is present.
In `@web/src/data/leaderboard.json`:
- Around line 83-86: Regenerate the leaderboard gate data from the frozen lane
and current model registry: update frozen_output_token_cap to reflect the
4,096-token policy and replace the obsolete embedded three-model list with the
current registry entries. Preserve the empty missing list and
model_registry_frozen state, and apply the same regenerated values to the
additional leaderboard records noted in the comment.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: d144199d-9889-4b95-a46f-05ca95c7c6d0
📒 Files selected for processing (27)
README.mdconfig/openrouter_pricing_snapshot.jsonconfig/output_budget_sweep.jsonconfig/publication_protocol.jsonconfig/sota_v2_lane.jsonconfig/sota_v2_models.jsonconfig/sota_v2_smoke_manifest.jsondocs/PUBLISH_READINESS.mddocs/output_budget_sweep.mddocs/run_logs/sota-v2-panel-revision-2026-07-16.mddocs/run_logs/sota-v2-smokes-2026-07-16.mddocs/run_logs/sota-v2-smokes-2026-07-17.mdgm_bench/cli.pygm_bench/official.pygm_bench/providers.pygm_bench/publication.pyresults/analysis/output-budget-cost-estimate.jsonscripts/analyze_publication_panel.pyscripts/estimate_publication_cost.pyscripts/run_publication_matrix.pytests/test_publication.pytests/test_publication_analysis.pytests/test_publication_cost.pytests/test_publication_runner.pytests/test_smoke_manifest.pyweb/scripts/build_leaderboard.pyweb/src/data/leaderboard.json
💤 Files with no reviewable changes (1)
- tests/test_publication_cost.py
| "selection_policy": "User-curated current frontier panel: five large American-lab proprietary models and five leading open-weight models. Every OpenRouter route pins an exact provider slug, endpoint tag, and dated endpoint name with fallbacks disabled. Models run at native-minimum reasoning: disabled when optional and the lowest catalog-supported effort when mandatory. Kimi K3 is parked after its clean smoke failed and demonstrated cap pressure; the exact Nemotron 3 Ultra and DeepSeek V4 Pro routes are parked after bounded HTTP 404 infrastructure failures.", | ||
| "models": [ |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
The frozen panel contains ten models instead of the declared twelve.
The registry and required_smokes define five proprietary plus five open-weight models, while the publication objective requires five plus seven. Freezing this state propagates the wrong cohort into smoke completeness, cost planning, and leaderboard eligibility.
Also applies to: 181-199
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@config/sota_v2_models.json` around lines 14 - 15, The frozen model panel in
the selection_policy and models registry must contain twelve models: five
proprietary and seven open-weight. Update the model entries and associated
required_smokes coverage so the declared cohort, smoke completeness, cost
planning, and leaderboard eligibility consistently use the required
five-plus-seven composition.
| "id": "openrouter-tencent-hy3-free-novita", | ||
| "provider": "openrouter", | ||
| "model": "tencent/hy3:free", | ||
| "transport": "gateway-api", | ||
| "cohort": "open-weight", | ||
| "upstream_provider": "Novita", | ||
| "upstream_provider_slug": "novita", | ||
| "endpoint_tag": "novita", | ||
| "endpoint_name": "Novita | tencent/hy3-20260706:free", | ||
| "catalog_expiration_date": "2026-07-21", | ||
| "reasoning_policy": "disabled", | ||
| "reasoning_effort": null, | ||
| "fixed_options": { | ||
| "OPENROUTER_REASONING_ENABLED": "false", | ||
| "OPENROUTER_JSON_MODE": "false" | ||
| }, | ||
| "absent_options": ["OPENROUTER_REASONING_EFFORT"], | ||
| "role": "Tencent open-weight frontier anchor on OpenRouter's temporary free Novita route; prompt-only JSON because the endpoint does not advertise response_format" |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Do not freeze a route that expires on July 21, 2026.
This route expires only four days after the July 17 freeze. Publication execution or reproduction after that date will fail unless the run is explicitly bounded before expiry or a stable replacement is selected and smoked.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@config/sota_v2_models.json` around lines 161 - 178, Remove the expiring route
entry identified by id openrouter-tencent-hy3-free-novita, or replace it with a
stable, smoke-tested route that remains valid beyond the freeze. Do not retain
this route unless the publication process is explicitly bounded before its
catalog_expiration_date.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/run_publication_matrix.py`:
- Around line 803-806: The mandatory-reasoning validation in
gm_bench.publication.smoke_manifest_issues must reject reasoning_tokens when it
is not an integer, is a boolean, or is negative, matching _record_smoke_issues
before allowing panel unlock. Update scripts/run_publication_matrix.py lines
803-806 accordingly; add coverage in tests/test_publication_runner.py lines
570-599 through the persisted-manifest or build_cells("panel") path for True and
-1.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 05900922-07e3-4c75-8a85-3b87b2202250
📒 Files selected for processing (10)
config/openrouter_pricing_snapshot.jsondocs/PUBLISH_READINESS.mdresults/analysis/output-budget-cost-estimate.jsonscripts/estimate_publication_cost.pyscripts/run_publication_matrix.pytests/test_publication.pytests/test_publication_cost.pytests/test_publication_runner.pyweb/scripts/build_leaderboard.pyweb/src/data/leaderboard.json
🚧 Files skipped from review as they are similar to previous changes (3)
- results/analysis/output-budget-cost-estimate.json
- config/openrouter_pricing_snapshot.json
- docs/PUBLISH_READINESS.md
| if entry.get("reasoning_policy") == "mandatory-minimum" and ( | ||
| not isinstance(reasoning_tokens, int) or isinstance(reasoning_tokens, bool) or reasoning_tokens < 0 | ||
| ): | ||
| issues.append("artifact is missing reasoning-token telemetry for a mandatory-reasoning model") |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Apply the reasoning-token invariant at the panel-unlock gate.
_record_smoke_issues now rejects boolean and negative values, but smoke_manifest_issues still accepts them for mandatory-reasoning models. An edited invalid manifest can therefore unlock the panel.
scripts/run_publication_matrix.py#L803-L806: apply this same predicate ingm_bench.publication.smoke_manifest_issues.tests/test_publication_runner.py#L570-L599: exercise the persisted-manifest orbuild_cells("panel")path withTrueand-1.
📍 Affects 2 files
scripts/run_publication_matrix.py#L803-L806(this comment)tests/test_publication_runner.py#L570-L599
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/run_publication_matrix.py` around lines 803 - 806, The
mandatory-reasoning validation in gm_bench.publication.smoke_manifest_issues
must reject reasoning_tokens when it is not an integer, is a boolean, or is
negative, matching _record_smoke_issues before allowing panel unlock. Update
scripts/run_publication_matrix.py lines 803-806 accordingly; add coverage in
tests/test_publication_runner.py lines 570-599 through the persisted-manifest or
build_cells("panel") path for True and -1.
What changed
Revised panel
Proprietary: GPT-5.6 Luna, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Muse Spark 1.1.
Open-weight: GLM 5.2, Kimi K3, Nemotron 3 Ultra, MiniMax M3, Qwen 3.7 Plus, DeepSeek V4 Pro, Mistral Medium 3.5.
Terra, Sol, Claude Opus 4.8, and Fable 5 are recorded as future candidates.
Routing and reasoning
minimal; Grok useslow; Kimi K3 usesmax, its only catalog-supported effortamazon-bedrock/globalbecause the direct Anthropic route was unhealthy during selectionAll twelve no-spend live endpoint/credential preflights passed.
Cost plan
Refresh the estimate from accepted smoke telemetry before approving the full panel.
Validation
443 passedworkers=1Summary by CodeRabbit
New Features
Updates
Documentation